Optimal timing of audio-visual text presentation: the role of attention

This study investigates the optimal timing of audio-visual presentation for text comprehension. In Experiment 1, participants were asked to read and/or hear the expository passages in three conditions; visually presented, auditorily presented, and audio-visually presented condition. Comprehension performance in audio-visual presentation condition was not different from that in visual or auditory condition, raising the possibility that cognitive load on processing both visual and auditory information negated the positive effect of multi-modal presentation. To reduce the load of processing dual information simultaneously, we proposed to delay the timing of one of the two modality presentation and to direct participants’ attention to one of the two modality information during audio-visual text presentation. In Experiment 2 to 4, passages were presented audio-visually in three conditions; auditorily preceding, simultaneous, and visually preceding conditions. Participants were instructed to direct their attention to whole information (Exp. 2), visual information (Exp. 3), and auditory information (Exp. 4). The results showed that comprehension performance was higher in the visually preceding condition when their attention was directed whole or visual information. Based on the results, the integration process of audio-visually presented text information was discussed.